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@ Advanced user interface. 

^ An advanced user interface for use with a computer system operating on an integrated operating erwiron- 
SenfThe tntecrateroperating environment allows a plurality of application programs to be runnmg s.multa- 

transmitted and send the corresponding command to that application program. 
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This invention relates to improvements in entering input data into digital '^^'^^^''[^■^^'IJ'^''^^ 
relates to an advanced user interface which allows a user to select one or more .nput devices to mput data 
into a computer running a program originally written for a different input device^ sophisticated 

in the past, computers were used only by scientists. nnamern« and other h,gh eve -OP^^^I 
computer users. As computer technology progressed, and particularly with the advent or p 

*«^a7d,spite the prevalence of c<»T,pule,e in our society, men, ere apprehensive about "»"a 
The rdZafSlilpie and atphanumeHc keyboard ^^''^'^1:^0:^'' ^^iT^ J^^Tl 

for the unsophisticated user who is likely to avoid the effort altogether. friendly" 
It has therefore become necessary to design what have become known '"J^f/^^J-^^^^^^'i''^ 

°" SL^aTfacTsLdU^vV sh^^^^^^^^^ a device which allows the user to input data directly on the v^ua. 
overlay which consists of two transparent conductor planes d.sposed o" a surface^ 

, cr:r;'zrr^::s1rrr^^^^^ ^:;,rorrn 

screen, which includes transparent conductors dr.ven by ^" alternately, a stylus is 

detect the location of a finger touch by 'n c^^^^^^^^^^^^ ^oCte'io re^in'e the'stylus 

used to return the electromagnetK: s-gnals f -^J*^^ /^J ^y,^^^^^^^^ display screen having a number 

='-^:t,rr rrd^^nrdirs^r^rr I'r: 0, .he such as voice reco,ni,on 

. "-r 'aS£rr:^^;rrres, des,..^^^^^^^^^^ 

solt«are. One technique «hich has been ^"-J^^ / '» '"J S no'ri^^uired to comn^it long 

ror^;rLn=r^^;ruc:r^^^^^ 

. ;rur':^rn.rrn::e;":ru:rann^rra?:^^^^ an alpLumenc character 

3 rr:;rsrer rnte... o, cc^ 

integration, the user must employ separate ^''^'^^ X ITe diS^^^^^^^^^^^ »« outputs 0. 

r dSrprr ~a r la^SS :r o'—rrn to ^.e a single mt^rated piece 
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of software which incorporates a variety of applications which Is called a multiple-function program. 
Typically, these multiple-function programs include text, spreadsheet, and business graphing applications. 

Another approach is to provide an integrated operating environment as implemented by Microsoft 
WINDOWSTM or IBM Presentation Manager™. In this approach, individual application programs share 

5 information and often appear on the display screen at the same time, each in its own window. By selecting 
the strongest individual application programs, a much more powerful environment can be tailored for the 
user's particular needs, in contrast to multi-function programs where the user is limited to whatever 
applications have been programmed into the package. 

Unfortunately, while many user friendly input devices and many user friendly computer programs have 

10 been developed, these efforts have not been well integrated. For example, there are a targe number of 
programs which have been written to accept keyboard or mouse input which do not recognize information 
from a touch input device or voice recognition unit. This situation presents a severe obstacle to the 
introduction and widespread use of new user friendly input devices. Either a programmer must make 
extensive revisions to a large body of existing software, or must write a new multiple function program, or 

75 set of application programs each time a new input device is developed. Either alternative adds significantly 
to the costs and time of a development effort and may be beyond the capabilities of the organization 
developing the input device or render the effort too expensive for the expected benefits. In addition, 
copyrights or other proprietary rights may prevent the input device developer from adapting existing 
software. Further, newly developed software may not be as efficient or commercially acceptable to users as 

20 the established application programs already written for a particular purpose. 

A final consideration is that the individual user is probably best suited to assessing his own needs, yet 
is least likely to be able or interested in undertaking extensive revisions in application programming. It 
would be advantageous to allow a user to select the particular input devices and application programs which 
would best serve his purposes and yet with only a relatively low level of expertise, allow him to assemble 

25 the components in a user friendly interface between himself and the computer. The general solution 
developed by the programmer may not be optimum for a particular user's needs. It would also be 
advantageous to assure a user that future advances in input devices and application programs can be 
accommodated by the interface with a minimum of revision. 

It is therefore an object of the invention to allow a user to select between a plurality of input devices to 

30 input data into a computer system. 

It is another object of the invention to input information to a computer system via a different input 
device than that for which the active application program was originally written without revising the 
application program code. 

It is still another object of the invention to allow a relatively unsophisticated user to select among 

35 available application programs and input devices and construct a user interface designed particularly to his 
needs. 

It is yet another object of the invention to accommodate future advances in user fnendly input devices 
and application programs in a user interface. 

It is still yet another object of the invention to allow different users who operate a single computer to 
40 construct different user interfaces to accommodate individual preferences in inputting data. 

These objects and others are accomplished by an advanced user interface which operates with an 
integrated operating environment capable of running a plurality of application programs and which utilizes 
the message and focusing functions provided by the environment. The advanced user interface is 
comprised of three major modules, an alternative input subsystem module, an interface profile module, and 
45 an environment link module. The alternative input subsystem module provides communication between the 
attached user friendly input devices and the remainder of the advanced user interface as well as application 
programs through the integrated operating environment. The alternative input subsystem module receives 
the input signals generated by the input devices and translates them to input messages useable in the 
advanced user interface. 

50 The interface profile module comprises a set of application profiles, one for each application program 
for which the user wishes to use an input device in a different manner from the way in which the application 
program was originally written. An entirely new input device may be used with the application program, or 
optionally, the user may change the input signals of the existing input device from those to which the 
application program originally responded. Each application profile contains a list of input messages mapped 

55 to corresponding commands useable by the application program. The interface profile module may also 
comprise a set of user profiles for each user who utilizes the computer system. The user profile contains a 
list of input messages and their corresponding commands which can be common to several application 
programs. 
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The environment link module communicates with integrated operating environment and contains 
alQorithms for the execution of the advanced user interface procedures. For example, .t takes the input 
mes ges generated by the alternate input subsystem, queries the integrated operajng 
Teh applfcation program is active, matches the input messages to the <=°^^«P°f "9^°'^^^,^ 
appropriate application profile within the interface profile module, and initiates an acton ^ased on the set of 
Suctions such as sending messages to the active application program via the °P«^^'"9 ^"";°"'"^"*^]^'! 
advanced user interface may also include a set of utility programs such as a pop-up keyboard or an image 

magnifier^utility^ a block diagram showing the interaction of computer syster. software and hardware in 
, response to actions of user using an integrated operating environment such as J^"^^^^^^^ 

FIG. 2 is an architectural block diagram of a computer system ,n accordance "^fj^l^'^^^''^ 
invention depicting the memory having read therein a series of application and operating system programs^ 
" ' f^ 3 is a block diagram showing the interaction of computer system software and hardware m 

response to actions of a user designed in accordance with a first embodiment of the P'«s«"V'"^«"*'°"-^^^„ 
, response^to^actions^^ 9^^ ^^^^^^.^^ ^.^^^ embodiment of the present invention in response 

to a voice input signal. 

P,r61s^''rk^Tag^rSrgr'rnteraction of computer system software and hardware in 
response to actions o? a user designed in accordance with a second embodiment of the present inven^on. 
, '"''°"nG 7 is a ,°ow diagram of the operation of the second embodiment of the present invention in 

"^'^°"^G^^^^f.^w Igim of the operation of a computer module which distinguishes between touch 
inout meant to emulate a mouse pointing device and touch input meant to be a gesture or character. 
' FIG 1 sJows the interaction of the user and the computer system utilizing application programs A. B. C 

c anH n «;n-R'? in a tvoical inteqrated operating environment such as Presentation Manager 56. 

Th! use inpl data So the computer system via device A (keyboard) 32. device B <'"o-f > or 
touci sense 36 AS the user inputs data, the selected device generates an interrupt to activate the 
approp itr river, the case of the keyboard 32. driver A ^0. -tivate<.: In the ^^^^^^ 
HrLr R 62 Is eneraized' and input to the touch sensor 36 activates the touch driver 63. The appropriaie 

,0 dr ve slnal the Sied operating environment 56. The integrated operating environment 56 keeps track 
Of wh c^app l ation program fs active and sends all input data to that ^^^^-^^^'^^Y^^Z^^^^ 

. "Luci-a function" as the integrated operating environment 56 determines the focus o»f ^ user at a 
* spSshS. »ro^lnlcate wi,. omo, applicaion. «, o, m.s=.9«s .« handled by me .ntegrated 

™.se ITlZ^^^^of^e'k oJ, be developed to communicate coheranlly wUK «,e Integrated 

so rZsTde:L c^egoTes :?'^ch commanda such aa handwriting ,acognl»on or S-'-"^^"' 
M ea°racSmmo7a,ed by tnis method, aapecially b, an application program such « 'P'^";"''!'" 
^ich as ™ understandlng'o. what the .yf«s o. touch commands Z^Z^^^ 

interaction between a user and a <=°'^P"^^^ ^V^^ PS/aT-T however the principles of this invention can be 
for use with a personal computer such as the IBM P&/«iiM, nowever, uic h k arrhiteetural 
extended to oLr individual workstations or to much larger data processing systems. The architectural 
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block diagram of FIG. 2 includes a central processing unit (CPU) 20 connected by means of a system bus 
22 to a read-only memory (ROM) 24 and a random access memory (RAM) 26. Also included in tfie 
computer system in FIG. 2 are a display 28 by which the computer presents information to the user, and 
plurality of input devices including a keyboard 30. and devices A 32 and B 34, which for the purposes of 
illustration in the following description are a second alphanumeric keyboard 32 and a mouse 34. but which 
could be a specialized keyboard such as a numeric keyboard, and a remote pointing device or a vanety of 
other input devices. The computer system further comprises a touch sensor 36 or touch Input device for 
use at the surface of the display 28. a voice sensor 38 and an image sensor 40. The I/O 42 can be 
connected to communication lines or other i/0 equipment. 

The random access memory (RAM) 26 includes application program A 50, application program 8 51, 
application program C 42. and application program 0 53. Examples of typical application programs would 
include word processors, spreadsheets, graphic programs, electronic mail, and data base programs. The 
RAM 26 also stores an operating system 54. such as DOS™ or OS/2™, and an integrated operating 
environment program 56, such as WINDOWS™ or Presentation manager™, which allows several applica- 
tion programs to be running concurrently. Other software includes driver A 60 and driver B 62. which 
interpret the electrical signals generated by the second keyboard 32 and the mouse 34 respectively for the 
operating environment 56. The touch driver 63. the gesture recognition unit 64 and character recognition 
unit 65 handle input signals from the touch sensor. Input signals from the voice sensor 38 and the image 
sensor 40 are processed by the voice recognition unit 66 and image recognition unit 68 respectively. 
Although depicted in FIG. 2 as entirely software, driver A 60. driver B 62, touch driver 63, gesture 
recognition unit 64, voice recognition unit 66 and image recognition unit 68 can also be a combination of 
hardware and software. Finally, also resident in RAM 26 are the set of software modules which make up the 
advanced user interface 100. 
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FIG 3 shows a computer system designed in accordance with the present invention. Rather than being 
connected to the integrated operating environment 56 via a standard device driver at least one of the 
plurality input devices is connected to the advanced user interface 100. via a recognition unit. In FIG. 3. the 
voice sensor 38 and image sensor 40 are connected to the advanced user interface (AUI) 100 via their 
respective recognition units, voice recognition unit 65 and image recognition unit 68. A keyboard 32. and a 
mouse 34. remain connected to integrated operating environment 56 via their respective dnvers driver A 60 
and driver B 62. since many application programs recognize keyboard and mouse input. Finally, in FIG. 3, 
the touch sensor 36 is connected to both the integrated operating environment 56 via the standard touch 
driver 63 and to the advanced user interface 100 via the gesture recognition unit 64 and the character 
recognition unit 65. The touch driver 63. gesture recognition unit 64 and character recognition unit 65 
translate signals from the touch sensor 36 to produce gesture or character "tokens" respectively. The series 
of positions of a finger or stylus on the surface of the touch sensor 36 can be recognized as handwritten 
alphanumeric characters by the character recognition unit 65. or as a gesture such as a circle of right hand 
arrow by the gesture recognition unit 64. The voice recognition unit 66 digitizes the sound received by the 
voice sensor 38 and performs pattern matching against previously defined sound patterns to produce voice 
"tokens". Image recognition unit 68 processes information from the image sensor 40 and outputs image 
"tokens" A "token" is an input signal which represents a series of electrical signals collected from an input 
device For example, the series of points collected when a user draws a circle on the touch sensor can be 
defined as the "circle token". Coupling the touch sensor 36 to integrated operating environment 56 allows 
the user to take advantage of application programs which are specifically written for the touch sensor 36 or 
other existing application programs for which only primitive mouse input signals are needed. Yet as the 
touch sensor 36 is also connected to AUI through the gesture recognition unit 64 and the character 
recognition unit 65. application programs which do not recognize touch input can still accept gesture or 
character "tokens" due to the translation capabilities of the advanced user interface 100. 

m other embodiments, the keyboard 32 and mouse 34 to be connected to both the AUI 100 and 
integrated operating environment 56 as is the touch sensor 36. In still other embodiments, all input devices 
are attached directly to the AUI 100 alone. By using a keyboard AUI link, a user could change the normal 
typewritten commands for an application program to those which he finds more convenient to input or 
memorize. As the AUI 100 is written as an application program, it is able to use the communication and 
focusing capabilities of the integrated operating environment. 

Another preferred embodiment of the present invention uses a single pointing device dnver (not 
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pictured) to combine the function of driver B 62 and touch driver 63 by accepting input frona both mouse 34 
and touch sensor 36 as well as any other pointing device. This embodiment is particularly adapted to 
operate with the touch input device described in U. S. Patent Number 4.868.332, to E. Greamas. et aL 
entitled "Combined Finger Touch and Stylus Detection System for Use on the Viewing Surface of a V.sua 
5 Display Device", filed June 26, 1986. or the remotely sensed stylus described in commonly assigned patent 
aoDlication Serial No. 264,409. filed October 28. 1988. entitled "Remotely Sensed Personal Stylus . by E. 
Greanias. et al.. which are hereby incorporated by reference. The pointing device driver arbitrates between 
mouse input from the mouse 34 and finger and stylus input from the touch input device 36 and sends 
aopropriate input messages to the integrated operating environment 56. ^ , t»,» 

w Referring back to FIG. 3. the AUI 100 is further divided into several computer code modules^ The 
environment link module 101 contains algorithms for the execution of the procedures of the AU 100 and 
provides communication capabilities to the integrated operating environment 56 The Alternative Inpu 
Subsystem (AIS) 103 provides an interface description which allows the designers of {"^"'^'y '"^^^ 

devices e.g.. voice sensor 38. to seamlessly interconnect with the Advanced User Interface 100. The AIS 
,5 103 translates the input signals sent by the input devices and their respective recognition ""fsjo '"P^^ 
messages useable by the remainder of the Advanced User Interface 100. A part of each '"P"* /^f^^^S^; 
such as a prefix GS/ to denote a gesture, will be devoted to identify the input device from which the nput 
signal is received. In this embodiment, the AIS 103 also time stamps each input message so that the 
correct application program receives the appropriate commands. 
.0 The interface profUe module 104 is comprised of sets of application profiles 105 and the user profiles 
107 which are files which list input messages produced by the AIS 103 from the input signals received by 
nptit devices 36, 38, 40, mapped to keyboard, mouse or other commands which are usable by exis^ng 
aop lca«on programs, e.g.. mouse clicks, keystroke messages. MACROS, utility programs etc. The 
Svfncrd UseMPterface ut ity (AUI utility) 109 is a set of utility programs such as a pop-up keyboard or an 
2S fmagrmagr^'er Utility which can be used with AUI 100. As shown, the AU. utility 109 can be -tten as 3 
separate application program, or set of application programs, so that it can use f^^^^^^^^^^^'^""^ 
capabilities of the integrated operating environment 56 to communicate with °»';^^/PP''"*'°" Pj^^^^^f^,, ^ 
A few examples of utilities which might be found in the advanced user interface utilit.es 109 include a 
••nop-up" keyboard, a "pop-up" numeric keypad, an image magnifying utility and an electronic mail routing 
30 sEp The -^op up" keyboard and "pop-up" keypad are functions which are preferably designed to operate 
w th a touch fnput device at the display surface. By issuing the proper command e.g., a circle gesture o 
to ching a keyboard icon, the keyboard or keypad will "pop-up" on the screen. A user can ^u-ckiy er^er 
columns of alphanumeric information directly on the screen by touching the keys ^'^P^^V/f ^^fj;,^^^^^^ 
thus eliminating the need to divert their attention to manual keyboard. The space needed for the keyboard 
35 or keyp^is quite small, and will not completely obscure the work area of the spreadsheet, -o;^ Jj^^c^ssj^ 
e c S is the active application program. The image magnifying utility will magnify a rectangle of fixed 
siz^around a point at whicE the appropriate gesture was made. The utility allows very -"urate positioning 
of a cursor in the expanded image. After stylus liftoff, the normal size display is restored, and the selected 
cursor coordinates are sent to the active application program. .ponnnse to a 

The flow diaqram in FIG. 4 depicts the operation of the computer system in FIG. 3 in response to a 
voice inpurslg^T'e active application program is the spreadsheet 50. and presented to the user on he 
di 28 is an array of selected data concurrently with a menu bar of possible 

use Rather than striking a series of keys on the keyboard, the user elects to input data into the computer 
svSem us^ng another input device, the voice sensor 38. In response to the user vocalizing the word 

.5 -graph" a? 111 the vdce sensor 38 generates electrical signals and sends them to the appropriate 
tran^tion device the voice recognition unit 66. In response to these electrical signals, the voice recognition 
unit is activated at^l^ and sends an appropriate input signal, a voice input "token", to the Alternate Input 
Sui ys^em To3 At 1 15, the AIS 103 forwards an input message to the environment link 101 which contains 
'format on on the time the input signal was received and which input device sent the -Sna^^^ln respo^^ to 

,n he incut message at 117 the environment link 101 queries the integrated. operating environment 56 as to 
whi atTcatfon program is currently active. After determining the active application program, spreadsheet 
50 a 119 the environment link 101 reters to the applicatfon profile 105 of *►^^=P:ff J°J,^^^^^^^^^^ 
command which corresponds to the input message "GRAPH" which will be recognized by the sp eadsheet 
50 i e . the menu selection -graph-. The environment link 101 then at 121 "^^^f f^f ^^^^^^^^ ^^^^^^ 

55 determine whether there is a higher priority command which would ^''^'^l^l^^'T^'^X^^ 
application profile 105. The environment link 101 determines which command has higher P"orty ^t 123. 
The eniment link 101 at 125 then sends the higher priority command, .n tt^'^^;;!;;;P'^;;^3^3 
selection command from the application profile 105. to the integrated operating environment 56 which 
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routes the command to the active application program spreadsheet 50. The spreadsheet 50 at 127 takes the 
appropriate action in response to that command, in this case, graphing the cun-ently selected data and 
changing the information presented to the user by the display 28 accordingly. 
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To understand the power and flexibility of the advanced user interface 100. the application profiles 105 
and the user profiles 107 of the interface profile 104 must be explained in greater detail. Both types of 
profile associate lists of input messages with a list of corresponding commands which are recognized by 
the appropriate application program, the environment link module 101. or by the integrated operating 
environment 56 itself. The command can be the name of a program, an instruction recognized by the 
integrated operating environment or a MACRO. A MACRO is a file which contains an arbitrarily long string 
of "events" and/or utilities. An "event", for the purposes for this specification, is an input message which 
results from a user action such as a single keystroke, a mouse button press, a mouse button release, a 
mouse double click, single menu selection, a single gesture or other "token", or a single accelerator key 
selection. An example of a utility is the pop-up keyboard. A MACRO may also treat other previously defined 
MACROS as commands. Accelerator keys are often a feature of the integrated operating environment 56. 
They specify a keystroke sequence which emulates a menu selection. 

For example, an application may specify that the DELETE key is the same as selecting CUT from the 
EDIT menu and that SHIFT'INSERT is the same as selecting PASTE from the EDIT menu. These two 
examples are. in fact, widely used accelerators. 

Each application program may have its own application interface profile 105. which contains mappings 
of input messages and their corresponding commands for each input device connected to the alternate 
input subsystem 103. For example, one application profile might contain input messages and commands as 
shown in FIG. 5. The sample application profile shows input messages mapped against the corresponding 
command together with comments for display to the user describing the user action needed to create the 
input message and comments describing the action the computer system will take upon receiving that input 
message when the given application is active. 

The same input message from the AIS 103 will generally invoke a different MACRO on different 
application profiles 105. For example, a certain application profile 105 may specify that a MACRO named 
"Cut/Paste" should be executed in response to a Right-Arrow gesture, while a different application profile 
105 may map the Right-Arrow gesture to the keystroke sequence "Form 201 (ENTER)". When the gesture 
recognition unit 64 detects a right arrow gesture from the touch sensor 36. it calls environment link 101 
which determines the application program which owns the currently active window. The environment link 
101 reads the corresponding application's profile 105. merges that information with the appropnate users 
profile 107. choosing the command with higher priority, then issues the command to the active application 
program The integrated operating environment 56 or to other AUI modules as appropriate. 

The user profile 107 contains commands which are common to all or several of the application 
programs 50-53. A separate user profile 107 can be written for each system user. The advanced user 
interface can determine the active user during log-on procedures. Entries in the user profile 107 can be 
used to override those in the application profiles 105. This situation might occur when a particular user 
prefers to input data in a different manner from that provided by the application profile 105 written for a 
group of system users. The user profile 105 can also be used to provide a global default action, i.e.. if a 
particular input message was not found by PM-Link 101 in the appropriate application profile 105. Where a 
user profile 107 is used for a global default action, the application profile 105 which has a MACRO for a 
given input message can be given a higher priority than the default action. 

While the preferred embodiments of the present invention make a distinction between the application 
profiles 105 and the user profiles 107, in other embodiments, it is possible to write separate application 
profiles 105 for each user. Where the interface profile module 104 was organized in this manner, if an 
application profile 105 did not exist for a particular user, the environment link 101 would refer to a default 
application interface profile 105. The default application profile 105 would be provided with the application 
program the input device or by the programmer configuring the computer system. As currently envisioned, 
an application profile 105 for a given application program would generally be provided with the application 
itself Alternatively, all application profiles 105 as well as a general user profile 107 could be provided by 
the system programmer. For an unsophisticated user who does not wish to explore the power of the 
advanced user interface 100. the default profiles will usually be adequate. However, many users will wish to 
tailor the advanced user interface 100 to their particular desires and specification. Therefore, it will be 



8 



BNSOOCII><EP 089«1*A2J.> 



EP 0 394 614 A2 



10 



15 



necessary to either write or modify an existing application or user prof. e. 

The Advanced User Interface Control Panel, one of the Advanced User Interface ^t" 't'es 1 09. anows tne 
n,Pr tcTta^or r^anv of the AUI functions to create a user unique profile in an extremely efficient and user 

c^Tyl^ nrMSoughr advanced user interface 100 contains fairly sophisticated co.^^^^^^^^^^ 
hrough the use of the Control Panel the user need not understand 't%-°'^'"9«- ^'^^^^^^^ 
oZd by the AUI Control Panel is the ability to modify or create profiles. ^^'S 'S a^^^^ 
the interface profile create/modify module of the AUI Control Panel by select ng J^^^LTZ this 
Control Panel menu. The Control Panel first queries the user the name of the prof, e ° ^J^^^^^^^^ 
poTnt a menu of the appropriate profile is displayed to the user, including a 'l^^^^^^^^^^^^^^ 
actions eq circle gesture, "cut" voice command, etc.. against a descnption of the action taken Dy tne 
actions e.g., circie 9J • jo the user action. If there is no application or user profile 

Tecord command function or the record MACRO function. For example, returning to the first item ,n FIG. 5, 
a single line of the menu for the spreadsheet application profile may read: 



20 



USER ACTION 


COMMAND 


CIRCLE gesture on touch sensor 


Invoke the pop-up keyboard 



H .h .s .0 Change ,h, inpu, -ssags .ro. c.de gesture .o l^'^'^l^^^^^'^, 

» P,estn«rLaTer^6 mere Jo .h.e. cooperating „».,es »hi=. a»ow me .s. ,o creare . 

"The M"^Xa,ru° iCSrs I PreSa.on Manager program »nic« can be ae, up a. a .u,o,i.^ 

macro ^,ore 'L:^^--- ,';:^'::J^::^ Z''T::L^i r«n"on,enJo. me Eng.s. 
commands in the macro. The moaiticaiion may " ♦,,rninn on the "record" function to overwrite 

calls the MEU, if necessary. 
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One example of a default user profile is described with reference to a touch input device substituting for 
a mouse pointing device. As many programs recognize mouse input, a great many application programs 
could be operated using this default profile. 

In this example, the advanced user interface can be set in one of a plurality of modes which determine 
what mouse equivalent commands are found in the user profile as a result of stylus or finger actions. When 
the AUI is in mode one no mouse button messages are generated by touchdown, but does generate mouse 
move commands while the Finger or stylus are moving in the touch panel. Upon lift-off of the stylus, a 
buttondown, buttonup command sequence is sent to the application program. Mode one allows accurate 
positioning of the cursor before action is taken by the active application. When the advanced user interface 
is in mode two, no mouse button commands are generated by touchdown, mouse move message are 
generated by moving the stylus, and a mouse double click is generated upon lift-off. When the AUI is in 
mode three, a mouse button down command is generated on contact and a mouse button up command on 
lift-off of the stylus. Mode three is useful for dragging screen objects (icons, etc), sizing windows and 
marking areas in a window's client area (e.g., marking text which is to be moved). These modes may be 
selected manually by touching the touch panel repeatedly at an icon which indicates which mode the AUI is 
in. or automatically by the application profile. 
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Second embodiment 

FIG 6 shows another embodiment of the Advanced User Interface, where Presentation Manager 156 is 
used as the integrated operating environment, and OS-2 154 is the operating system which runs the 
computer system. Because of the peculiarities of 0S,2 154 and Presentation Manager 156, a slightly 
different arrangement of the Advanced User Interface 200 is necessary. 0S,2 154 has a single queue 
module which timestamps all input and requires that all input be queued within this module in the order m 
which it was transmitted to the computer system. Therefore, all of the alternate input devices must go 
through OS.2 154 before being handled by the environment link module PM-Link 201. In addition, the 
present implementation of the single queue feature allows only keyboard and mouse input, and will not 
handle input from a touch input device, or other alternate input devices. 
30 These constraints lead to the advanced user interface design shown in FIG. 6. All input devices 
keyboard 130 device G 132, device D 134, device E 136 and device F 138. go through OS/2 154 to be 
added to the single queue of OS/2 154. The keyboard 130 goes directly through a standard device dnver 
140 to OS/2 154. Device C 132. device D 134. device E 135 and device F 138. go through device-specific 
code 141 142 143 and 144, respectively, to the alternative input subsystem 203, before being passed to 
35 the single queue of OS/2 154. Since the single queue will only handle keyboard or mouse input, the 
alternative input subsystem 203 converts the input signals from device C 132, device 134. device 136 and 
device 138 into mouse or keyboard input messages. In addition, since OS 2 154 will not handle touch, voice 
or image "tokens" as input messages, all such inputs are passed through as a series of mouse move points 
or other "events". 03 2 154 then passes the input messages to Presentation Manager 156 which keeps 
40 trackof the active application program. , 
In FIG 6 PM-Link 201, the environmental link module of the Advanced User Interface 200, acts as a 
filter between Presentation Manager 156 and the application programs. Application E 150, Application F 
151 Application G 152 and Application H 153. After receiving an input message from Presentation Manager 
156* together with information about which is the active application program, the PM-Link 201 refers back to 
the alternate input subsystem 203 to determine whether the "keyboard" or "mouse" message it received ,s 
in fact a keyboard or a mouse message, or rather a touch, voice, image, or other message. The PM-bnk 
201 then passes the true message to the appropriate application program. The remaining modules of the 
advanced user interface 200 are not called unless the active application program indicates that the true 
input message is not understood. If the active application indicates that the true input message is no 
understood. PM-Link 201 then refers to the interface profiles module 204 to determine the correct 
command. If the true input message indicates that it may be a gesture, character, voice or image token , 
the PM-Link 201 sends the input message to the appropriate recognition unit to identify the token, and then 
searches the interface profiles 204 to determine the correct command. 

The flow diagram in FIG. 7 depicts the operation of the computer system in FIG. 6 m response to a 
voice input signal. The active application program is the spreadsheet 150. and presented to the user on the 
display 28 is an array of selected data concurrently with a member of possible functions available to the 
user Rather than using the keyboard 130, the user elects to input data using another input device, the voice 
sensor 136 In response to the user vocalizing the word "GRAPH", the voice sensor 136 generates 
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electrical signals and thereby Invokes the appropriate device spec.f.c code 144 .s 221. The 

device spedfic code at 223 then sends a series of input signals to the AIS 203 corresponding to the word 
••GRAPH". The AIS 203 translates the input signals as a series of keyboard or mouse input messages to be 
added to the single queue of OS/2 154 at 225. After timestamping the stream of input messages. OS/2 
passes mem to Presentation Manager 156 which keeps track of the focus of the user and passes the input 
messages to the active application program, the spreadsheet 150. 

The PM-Llnk 201 Intercepts the message to the active application program at 227 refers back to 
the AIS 203. to determine that the true input messages are a set of voice input messages at 229 and sends 
the true input messages, the voice input, to the spreadsheet 1 50 at 231 . ,«cn„nH with 

After the PM-Link 201 transmits the true voice input message, the application program will respond with 
its own message at 233 and one of two things may occur. If the application does not ""^^^^^^"f f ^j^f ° 
nput message sent, it returns a "RO" message to the PM-Link 201 at 233. mdica ing that it does not 
understand L input message. The PM-Llnk 201. in response to an RO ^-^^^^^e. would normally go to ^ 
interface profiles 204 to determine the corresponding command in the appropnate application profiles 205 
,5 ^241 and user profile 207 at 243. However, since the input messages indicate that t ey -a a vo.ce 
messaae at 237 the PM-Link 201 first sends the input messages to the voice recognition ""'t 215 to 
Tecete the correct v^^^ input token for the word "GRAPH". After receiving this token at 239. the PM-Link 
201 L boks !n the appLtion profile 205 of the spreadsheet 150 at 241 to find the corresponding 
mmand to the input me'ssage which will be recognized by the spreadsheet 150 Le the -"u se ecj^^^^^ 
20 graph- Then the PM-Link 201 refers to the user profile 207 of the current "^^r at 243^The PM-Unk 201 
?hen determines which profile has the higher priority command for the spreadsheet f ^^^^^^^^^^^^ 
sends the higher priority command to spreadsheet 150 at 247. The spreadsheet then takes the ^PP'°P"f^ 
Son at 249 in this case, graphing the currently selected data and changing the information presented to 

3. ToXTo^TefZZue spreadsheet 150 can decipher, hand or is otherwise aware of vo,^^^^^^^^^^^ 
"Rl " message is sent to the PM-Link 201 at 233. The PM-Link 201 would then take no furthe act on for the 
current message While the spreadsheet 150 may be aware of voice input, it may also elect Jo ignore the 

„, Advances user Interlace 200. When PM-Unk «>. .ec*ss an '"P"' J-Jf 9/, """^nni,^:,,^^ 

whether the set of points is recognized as a gesture. If it is recognized as a gesture. PM-Link 201 sends the 
r Jtn thP ntr?2:e orotiles 204 to find the set of instructions to send to the active application program 

L":tr, Ti:::S:t:oZ."ZeT^^^ ZZe., o. . n.ouse »u»onup avan,. and *a, .ouch input 

tawea^ the »o wis of .oucn input. Tha ALancad Usar Intartaca 200 allows "'^V"."*"''?' 
aw mo" a to tha deslted position belcra the application Is notiliad of tha event by placmg all me po^B 

"H'itrreiTdirr^r:^^^^^^^^^^ 

TsS p srT«m™rcati to tt,a applica«on progran,. Hd-arJ. «,a uyt su^^^^^^^ 
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by the user in the user profile 207 or may be altered by the application program which is utilizing the touch 

'"''"ff on the other hand, the user does not stop at a particular point for the specified time delay period and 
instead lifts the stylus off the touch screen, the Advanced User Interface 200 selects the set of input points 

s as a candidate for character or gesture recognition. In other words, the time delay provides a window in 
time in which gestures can be made. During this time, the Advanced User Interface 200 will only allow 
gestures to be made. If the user waits for the time delay period before lifting off. the points collected will not 
be candidates for a gesture, but a mouse event. If the user lifts off before the time delay period expires, an 
attempt will be made to recognize the set of input points in the character recognition unit 21 1 or gesture 

ro recognition unit 213 as a gesture or character. If it is not recognized, however, the normal mouse emulation 
sequence will be generated: mouse buttondown. mouse buttonup at the last input point in the set. If it is 
recognized, the PM-Unk 201 refers to the interface profiles 204 to send the appropriate commands to the 
active application program. 

The recognition of a circle gesture drawn by a user on the face of the touch sensor 134 is described 

75 below with reference to FIG. 8. 

A user having determined that he wishes to invoke the pop-up keyboard utility in the AUI utilities 
module 209, draws a circle of the face of the touch sensor 134 and lifts the stylus off before the end of the 
set time delay The touch sensor 134 generates a series of interrupts to the device specific code 142 at 261 
which passes a set of input signals to the AIS 203 corresponding to the set of points m the circle at 263. 

20 The AIS 203 translates the input signals to mouse input messages to be handled by the single queue in 
OS/2 154 at 265 The input messages are sent to the active application program, the spreadsheet 150. by 
Presentation Manager 156. but are intercepted first by the PM-Unk 201 at 267. The PM-Link 201 then 
queries the AIS 203 and determines that the "mouse" input messages are actually touch input at 269. 

When PK/I-Link 201 discovers the first touch input point, usually a touch down in the sensor 134 it refers 

25 to the user profile 207 for the time delay period selected by the user at 271. At 273. the PM-Link 201 
collects the touch input points in its stroke buffer until the stylus liftoff event is detected. As the PM-Link 
201 collects the touch input points, each time the user pauses on the touch sensor within the stroke, 
initiating a "period of stability" at 275. the PM-Link 201 starts timing the period of stability to determine 
whether the time delay period has expired at 277. If the user moves before the time delay period has 

30 expired the PM-Link 201 continues collecting the touch input points in the stroke at 273. If. however, the 
period of stability exceeds the time delay, a mouse buttondown. mouse buttonup command is generated at 
281 In this example, the user does not pause before the stylus is lifted, and therefore, the stroke is sent to 
the character recognition unit 211 and gesture recognition unit 213 for processing at 279. If the touch input 
stroke had not been recognized as either a gesture or character, a mouse buttondown, mouse buttonup 

35 command would be generated at 281. o.,. . • ■ nn-, 

After being recognized by the gesture recognition unit 213 as a circle gesture, the PM-Unk 201 passes 
the circle gesture to the spreadsheet 150. The spreadsheet returns an "RO" message at 285. 'ndicating that 
the circle gesture was not understood. Since the circle gesture was not understood at 287. the PM-Unk 201 
refers to the application profile 205 for the spreadsheet 150 at 289 and finds no corresponding command 
,0 for the circle gesture. The PM-Unk 201 then refers to the user profile 207 at 291 which contains 0^e 
corresponding command "invoke the pop-up keyboard". As the command from the user ProfHe 207 has tfie 
highest priority at 293. the PM-Unk 201 sends this message to the AUI utilities module 209 at 295 and the 
pop-up keyboard is presented to the user on the display. 

While the present invention has been described with reference to specific embodiments, it wil be 
45 understood by those skilled in the art that changes in form and detail .may be made without departing from 
the spirit and scope of the invention. For example, while gesture and character recognition has discussed in 
terms of a touch input device, another pointing input device such as a mouse or a graphic tablet can be 
utilized to send gestures or characters to the Advanced User Interface. In addition, while specific mention 
has been made of particular application programs, integrated operating environments and input devices, the 
flexibility of the Advanced User Interface can accommodate both further advances as well as the existing 
implementations of these programs and devices. The embodiments presented are for purposes of example 
and illustration only and are not to be taken to limit the scope of the invention narrower than the scope of 
the appended claims. 
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1. An advanced user interface, for use with an integrated operating environment capable of running a 
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Plurality of application programs si— us.y j a co^^^^^^^^ 

^r;:^^S^::rS;^ -s. interna, in response to actions 

^farrnTe ■:;:sZJ^:::':r.o.u^e coupled to said input device which translates said input signals to 
rnteirpX'; Which contain mappings of said input messages against corresponding commands useable 

by at least one of said plurality of f 3^^^^^^^ subsystem, said interface profiles and said 
an environment link module coupled to sa.d a^em^ ^P^^ ^"°J^.,^j ^^^^ messages, match.ng 
integrated operating environment sa.d ^^'"^^T^^'l'^^^ said interface profiles for the 

said input messages against sa.d --"^^^''^^^ ;°Z;Ts^^^^^ were transmitted by said input device, 
application program wh.ch ^^^^^^^^^^^V^'^Vtran ^pr^^ computer module in said random access 
and sending said corresponding commands to an app.un 

memory; ---oonse to user actions different from those user 

said computer system performing ^ '^^^^^^ written without change to the application 

actions for which the active application program was originally wrn 

program. ^ ... . ^..j. . wherein said interface profiles comprise at least 

2. The advanced user interface as ^^^^^^^ J^'^^^^^^^^ containing mappings of input 

one application profile for each of sa.d P'^^^'^y °j J^^^^^^^ 

messages against corresponding instructions useable by said ^PPi^at^^^P .^9^^.^ .^^^^^^^ ^,,,,33 ,,rther 
useable by more than one of said plurality ^f''^Xrf^^Bre\o at least one of said corresponding 

instructions contained in sa.d user Pjof'te -s a d^/^^'J J .^^^^^^..^^ ^ the input message. 

keyboard. ^ ... ciaim 6 which further comprises a second plurality of 

7. The advanced user interface ^'^J^'^ifonment through standard device drivers, wherein 

application programs were originally written. ^ interface in combination with an integrated 

9. A set of interface profiles for use '''''' r!To^ll^^^o^ programs simultaneously on a computer 
operating environment capable of r"""'"^; P'"^^^*^ Jai'^set Sf interface profiles comprising mappings of 
system only one of which is active at a gr.en t^^' ^^jf ° ^ j^^^^y ^f input devices coupled to said 
input messages derived from the input S'S^als of at leas^ one 0 P ^^^^^ ^^.^ ^^^^^^^^^ 

advanced user interface -^^^^^^ ^f'""^ ^ZZ'^^ Z^ P'°9^^'"^ ^^'^ ''''Tit' 

« of application programs so if response to ule^a^^^ from user actions for which the 

:^rn ^pC: wfHiinaly written wit^ut chang^^ to , one application 

10. The set of interface profiles as recited in C'^'"^ ^ j^gg of input messages derived 
p.ofile for each of ^fJS'^Z^S^ S:^^^ ^^-"^^ '"^'^^^ 
:g -nrcorpoXtc^mmands ^^^^^^^^.^ comprise a user profile which 

11. The set of interface profiles as recited p^^ ^^ , ^, ^ne of said plurality of input devices 
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messages, interface profiles which comprise mappings of said input messages against corresponding 

commands by at least one of said plurality of application programs, comprising the steps of: 

receiving input signals transmitted from at least one input device in response to actions performed by a 

user; 

5 translating said input signals into an input message; 

determining which one of said application programs was active at the time said input signals were 
transmitted; 

finding a corresponding command to said input message in said interface profiles for said active application 
program; and, 

10 sending said corresponding command to an appropriate computer module in said random access memory; 
said computer system performing a given action in response to user actions different from those user 
actions for which said active application program was originally written without change to said active 
application program. 

13. The method of operating a computer system as recited in Claim 12 wherein: 

75 said receiving and translating steps are performed by said alternative input subsystem: 

said determining step is performed by said environment link module by querying said integrated operating 
environment; and 

said finding and sending steps are performed by said environment link module. 

14. The method of operating a computer system as recited in Claim 12 or 13, wherein said finding step 

20 comprises the steps of: 

finding a corresponding command to said input messages in an application profile for said active 

application; 

finding a corresponding command to said input messages in a user profile for said user operating said 
computer system; and 
25 determining which corresponding command has higher priority. 

15. The method of operating a computer system as recited in one of Claims 12 to 14. wherein said 
corresponding command is sent to said active application program and/or to a utilities module of said 
advanced user interface. 

16. A method of generating a command to a computer system in response to user actions on a surface 
30 of a touch input device comprising the steps of: 

collecting touch input points as a user moves a stylus across said touch input device surface; 
in response to said user pausing at a given point on said touch input device surface, starting the timing of a 
period of stability in which said user does not move said stylus from said given point; and, 
in response to said period of stability exceeding a predetermined period of time, generating a mouse 
35 command to said computer system at said given point. 

17. The method of generating a command to a computer system as recited in Claim 16 further 

comprising the step of: 

in the absence of a period of stability exceeding said predetermined period of time before said user lifts 
said stylus off said touch input device surface, recognizing said touch input points as a gesture input 

40 message. -7 ♦ ^t, 

18. The method of generating a command to a computer system as recited in Claim 16 or 17 further 

comprising the step of: 

in the absence of a period of stability exceeding said predetermined period of time before said user lifts 
said stylus off said touch input device surface, recognizing said touch input points as a character input 
45 message. 

19. The method of generating a command to a computer system as recited in one of Claims 16 to 18 
further comprising the steps of: 

finding a corresponding command to said gesture input message in interface profiles containing mappings 
of input messages against corresponding commands; and, 
50 sending said corresponding command to an appropriate computer module in a random access memory of 
said computer system. 
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@ An advanced user interface for use with a com- 
puter system operating on an integrated operating 
environment. The integrated operating environment 
allows a plurality of application programs to be run- 
ning simultaneously, one of which is designated the 
active application program to which all input data is 
directed. The advanced user interface allows a user 
to select among user-friendly input devices to op- 
erate any application program according to his in- 
dividual preferences without change to the applica- 
tion program code. The advanced user interface 
includes alternate input modules which translate the 
input signals transmitted from the various input de- 



vices into input messages useable by the rest of the 
interface. The advanced user interface also includes 
interface profiles which contain mappings of the in- 
put messages against corresponding commands 
useable by the application programs, the integrated 
operating environment or other modules of the ad- 
vanced user interface itself. An environment link 
module refers to the interface profiles and matches 
the input message against the corresponding com- 
mand for the application program active at the time 
the input signal was transmitted and send the cor- 
responding command to that application program. 
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